Execution

Time Planning

Make sure to provide enough time for each participant based on the results of your pre-test where you should have carried out the whole experiment with 2 or 3 subjects. Provide enough time for saving your data and potentially correct faulty recording settings.

Backup Procedures

Make sure to make a back-up of your data after each trial or at least each day. Think about how much data loss you can accept.

Checklists

Make checklists for (1) before and (2) after each trial and (3) at the end of each day. Checklist (1) before each Trial should contain reminders for each modality e.g. Cams running, Mics running, Logging running. Checklist (2) after each Trial should contain reminders to (1) Switch off all recording devices (mention them all!) (2) check audio (3) check video (4) check other data (5) perform back-up of the data. Checklist (3) at the end of each day to carry out back-up of your data if this has not been done after each Trial.

Enforce the usage of the checklist. Even simple things can be forgotten.

Meta Data Collection

Meta Data contains information for each trial what errors or problems occurred within the experimental procedure or within the data recordings. Collect meta information for each trial directly. Use a format which can be automatically processed (by e.g. validation scripts) and check if it was correct. Communicate the current meta data to all experimenters/wizards… esp. if the experimenter needs to know the current condition (e.g. if you have to hand certain questionnaires per condition to a participant).

Consistency

Minimise the difference between trials.

Before the study

• Experimenters and wizards need training
• Wizard reactions need to be tested and must not change between trials
• Explanations by the experimenter must not change between trials
• Scenario or setup changes between participants add variance to the results and can make the trials incomparable (the need for changes can be minimized by testing)

When the study is already running

Use a checklist to enforce that each step is actually done (see Checklists)

Automation

Automate as much as possible to prevent some errors:

• Automate experiment workflow where possible
• Automate data conversion tasks
• Automate statistics and plot generation (already for analyses during experimentation?)

Experimenter-related

• each participant should get an identical intro

Wizard-of-Oz specific aspects

Make sure that every wizard gets the same introduction. The GUI for the wizard has to be simple and must not allow accidental clicks. Initial training runs before recording the data help to warm up and prevent initial learning effects to ripple into the data set. Check the hardware setup, if the wizard is able to react in time (video buffers, lags, etc.) and provide headphones and a monitoring microphone in case speech is important. Make sure that participants do not know there is a wizard.